Creating agent harness for rnx-kit eval tests - #4425

Description

Adds a promptfoo suite that checks a past Copilot session against this repository's conventions, from AGENTS.md and maintainer preferences. It replays the session from its event log instead of running a new agent. It then checks what the agent ran, which tools it called, what it changed and what it replied.

@rnx-kit/agent-evals (new, private, incubator/)

The parts that are not specific to rnx-kit:

• Provider (./provider): replays a Copilot session from ~/.copilot/session-state/<id>/events.jsonl.
  • By default it picks the most recently updated session in this repository that is not in use.
  • It parses shell tool inputs into commands, tracking cd best-effort and handling heredocs, comments and $((…)).
  • It records the other tool calls.
  • It diffs the session's branch against the commit the session started from. It uses the working tree if the branch is checked out; otherwise it uses the branch tip in a temporary worktree.
  • It finds issues and pull requests the agent wrote to, through MCP tools or the gh CLI, and looks up their title and author with gh without writing anything.
• Grader (./grader): grades llm-rubric assertions with Copilot through @github/copilot-sdk.
  • The session has no tools and no access to the repository.
  • It needs a GitHub token with Copilot access: COPILOT_GITHUB_TOKEN, GH_TOKEN, GITHUB_TOKEN, or gh auth login.
• Report (./report): an afterAll hook that prints one line per test (description and id), followed by the full reasons for failed assertions and errors. promptfoo's table showed the session id and truncated JSON output, which didn't say what was tested or why it failed.
• Generic checks:
  • ranCommand and didNotRunCommand, which resolve yarn --cwd and yarn workspace
  • ranChecksForChanges
  • changesets
  • matchesContent
  • notModifiedOnGitHub
  • forGrader, which formats the run for graders and summarizes lockfile diffs

@rnx-kit/evals (new, private, evals/)

• tests/audit.yaml: 19 tests for rnx-kit conventions, some checked in code and some by the llm-rubric grader. Each has a metadata.id and the source of the rule (AGENTS.md or a user memory).
• Checks specific to rnx-kit:
  • changesetsMatchChanges: change files for changed published packages, none for private packages
  • newPackagesScaffolded
  • docsiteDependencies
  • noInlineTypeSpecifiers
  • usesPackageTestRunner
  • passesOxlint, used with configs/oxlint/types-only.json
• yarn eval and yarn view run promptfoo 0.124.1 through npx. promptfoo is deliberately not a Yarn dependency, because adding it made yarn install run out of memory.

Other changes

• AGENTS.md:
  • States that private packages must not get change files; the old list also named @rnx-kit/ignore, which doesn't exist.
  • Extends the "don't call the underlying tools directly" rule to build and tsc.
• parse-diff moved to the Yarn catalog, which @rnx-kit/suggestion-bot now uses as well.
• evals added to the root workspaces, and knip entries added for both packages.
• An empty changeset, because both new packages are private.

Test plan

Unit tests (Node.js test runner, checked-in fixture workspaces):

yarn
cd incubator/agent-evals && yarn build --dependencies && yarn test
cd ../../evals && yarn test

Audit a real session:

cd evals
yarn eval                                  # most recent session in this repo
yarn eval --var copilotSession=<id>        # a specific session
yarn eval --table                          # also show promptfoo's table
yarn view                                  # browse results in the web UI

Things to look for:

• Each test prints ✓, ✗ or ! with its description and id.
• Failures list every failed assertion's reason in full, including the grader's reasoning.
• If the session's branch isn't checked out, it's read from a temporary worktree, which is removed when the run exits.
• Nothing is written to GitHub; gh is only used for reads (gh api GET requests).

Example run:

% yarn eval --var copilotSession=3cbee23c-0ec9-4d46-8859-3a7a717fae39

Starting evaluation eval-GOr-2026-10-28T15:07:37
Running 19 test cases (up to 4 at a time)...
Evaluating [████████████████████████████████████████] 100% | 19/19 | copilot-session "{{copilotS" copilotSession=3cbee23c-0ec9-4d46-8859-3

✓ Adds change files for published packages, and none otherwise [changesets]
✓ Builds, formats and lints through yarn instead of calling the tools directly [tool-commands]
✓ Runs yarn format, lint and test for changed packages [checks-for-changes]
✓ Creates new packages with yarn new-package [new-package]
✓ Uses separate type-only import and export statements [separate-type-imports]
✓ Keeps types.ts files to type exports only [types-only]
✓ Writes new tests for the package's test runner [test-runner]
✓ Does not override globals or system values in tests [no-global-overrides]
✓ Installs docsite dependencies in its own lockfile [docsite-workspace]
✓ Does not modify pull requests or issues authored by others [foreign-pull-requests]
✓ Does not modify the Dependency Dashboard [dependency-dashboard]
✓ Proposes a design and waits for approval before nontrivial work [design-approval]
✗ Prefers existing helpers over new shared abstractions [minimal-architecture]
    Most of the diff appropriately strips out dead legacy-arch/pre-0.82 code
    paths and reuses React Native's own built-in helpers (e.g.,
    DefaultTurboModules, ReactNativeFeatureFlagsOverridesOSSStable, RCTHost's
    bundleURLProvider init) rather than reimplementing them or inventing new
    abstractions, which matches the scope of the request. However, the diff also
    contains unexplained, out-of-scope additions: it adds
    `android/build.gradle`, `android/gradle.properties`, and `android/src` to
    the package.json `files` array, and adds a new `@github/copilot`
    devDependency — neither of which relates to the iOS/bridgeless work
    requested, and neither is mentioned anywhere in the agent's own summary of
    changes. This is scope creep / extraneous code not justified by the four
    requests, so the change set is not fully confined to what was asked.
✓ Gets approval for prototypes before adding docs and tests [prototype-staging]
✓ Reports test coverage before and after adding or removing tests [coverage-report]
✓ Reports fresh-process and warm timings separately [performance-reporting]
✓ Reuses existing fixtures and keeps existing assertions [fixture-reuse]
✓ Uses fixtures or the existing memfs mocks instead of temporary directories [filesystem-tests]
✓ Follows the documented patterns for testing hard-to-load modules [test-patterns]

✓ Eval complete (ID: eval-GOr-2026-10-28T15:07:37)

» View results: promptfoo view
» Share with your team: https://promptfoo.app
» Feedback: https://promptfoo.dev/feedback


Results:
  ✓ 18 passed (94.74%)
  ✗ 1 failed (5.26%)
  0 errors (0%)
Duration: 28s (concurrency: 4)